{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Policy gradient intuition\n",
    "\n",
    "Policy gradient is one of the most popular algorithms in deep reinforcement learning.\n",
    "As we have learned, policy gradient is a policy-based method by which we can find\n",
    "the optimal policy without computing the Q function. It finds the optimal policy by\n",
    "directly parameterizing the policy using some parameter $\\theta$\n",
    "\n",
    "\n",
    "The policy gradient method uses a stochastic policy. We have learned that with a\n",
    "stochastic policy, we select an action based on the probability distribution over the\n",
    "action space. Say we have a stochastic policy π , then it gives the probability of taking\n",
    "an action $a$ given the state $s$. It can be denoted by $\\pi_{}(a|s)$ . In the policy gradient\n",
    "method, we use a parameterized policy, so we can denote our policy as $\\pi_{\\theta}(a|s)$ ,\n",
    "where θ indicates that our policy is parameterized.\n",
    "\n",
    "\n",
    "Wait! What do we mean when we say a parameterized policy? What is it exactly?\n",
    "Remember with DQN, we learned that we parameterize our Q function to compute\n",
    "the Q value? We can do the same here, except instead of parameterizing the Q\n",
    "function, we will directly parameterize the policy to compute the optimal policy.\n",
    "That is, we can use any function approximator to learn the optimal policy, and θ is\n",
    "the parameter of our function approximator. We generally use a neural network as\n",
    "our function approximator. Thus, we have a policy π parameterized by θ where θ is\n",
    "the parameter of the neural network.\n",
    "\n",
    "Say we have a neural network with a parameter θ. First, we feed the state of the\n",
    "environment as an input to the network and it will output the probability of all\n",
    "the actions that can be performed in the state. That is, it outputs a probability\n",
    "distribution over an action space. We have learned that with policy gradient, we use\n",
    "a stochastic policy. So, the stochastic policy selects an action based on the probability\n",
    "distribution given by the neural network. In this way, we can directly compute the\n",
    "policy without using the Q function.\n",
    "\n",
    "Let's understand how the policy gradient method works with an example. Let's take\n",
    "our favorite grid world environment for better understanding. We know that in the\n",
    "grid world environment our action space has four possible actions: up, down, left,\n",
    "and right.\n",
    "\n",
    "Given any state as an input, the neural network will output the probability\n",
    "distribution over the action space. That is, as shown in the following figure, when we feed the\n",
    "state E as an input to the network, it will return the probability distribution over all\n",
    "actions in our action space. Now, our stochastic policy will select an action based on\n",
    "the probability distribution given by the neural network. So, it will select action up\n",
    "10% of the time, down 10% of the time, left 10% of the time, and right 70% of the time:\n",
    "\n",
    "![title](Images/3.png)\n",
    "\n",
    "We should not get confused with the DQN and the policy gradient method. With\n",
    "DQN, we feed the state as an input to the network, and it returns the Q values of all\n",
    "possible actions in that state, then we select an action that has a maximum Q value.\n",
    "But in the policy gradient method, we feed the state as input to the network, and it\n",
    "returns the probability distribution over an action space, and our stochastic policy\n",
    "uses the probability distribution returned by the neural network to select an action.\n",
    "\n",
    "Okay, in the policy gradient method, the network returns the probability distribution\n",
    "(action probabilities) over the action space, but how accurate are the probabilities?\n",
    "How does the network learn?\n",
    "\n",
    "__We will discuss this in detail in the next section.__"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
